Repository navigation
Guard quotes of app text, and record the screenshot guidance A/B - #341
Merged
Merged
Conversation
Red teaming's fence step, checked: the marking is a field, so no fence exists. Every quote of device text in a message goes through %q, and every WebDriverAgent predicate refuses a name or label holding a quote or a backslash. quoting_test.go now fails on a quote written around a %s by hand; a planted one failed it, naming the file and line.
Measured on the Android 17 emulator with Sonnet 4.5, marking on, 100 trials across t15, t16, t28 and t31: 17 of 40 trials with a screenshot in both arms where the text tools suffice; t15 passes 8 of 10 against 7 of 10.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two red-teaming steps from the ROADMAP.
quoting_test.gofails on a quote written by hand around a%sin a message; a planted control failed it.app_screenshotdescription telling agents to read withapp_mapandapp_textfirst changed nothing. 100 trials on the Android 17 emulator, Sonnet 4.5, marking on: 17 of 40 trials with a screenshot in both arms where text suffices (p = 1.0); t15 passes 8 of 10 against 7 of 10.